Skip to content

Keep the worker alive through transient runtime failures while it registers (sdk-php#448 parity) - #355

Merged
rmcdaniel merged 4 commits into
mainfrom
general/worker-transient-resilience
Nov 20, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
general/worker-transient-resilience

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Python counterpart of durable-workflow/sdk-php#448: a worker process died when native Server answered 503 {"reason":"storage_unavailable","message":"storage_unavailable"} during warm-up.

What the audit found (origin/main cd5cff4)

I drove Worker.run() against a mock transport that answers 503 storage_unavailable on one route ten times and then recovers. These are the results on main:

Path Result on main
GET /api/cluster/info at startup Worker exits. Worker._register wraps the error as RuntimeError("Server compatibility error: unable to read /api/cluster/info: server returned 503 ...") once the client's 3 transport attempts (about 0.3 s) are used up.
POST /api/worker/register at startup Worker exits with ServerError(503 storage_unavailable).
Workflow, activity and query polls Keep running. _poll_*_tasks catch every exception and back off with jitter (_PollFailureBackoff); only storage-admission errors are raised.
Worker heartbeat Keeps running. _send_heartbeat logs the failure and returns None.
Task completion and failure reports Keep running. Workflow-task completion retries and then reports; activity outcomes are logged.

Change

Cluster discovery and registration are now repeated whole through transient failures, in both run() and run_until() (within its timeout). The wait follows the shared schedule uniform(0, min(4, 0.25·2^(n−1))) seconds, each retry logs a warning, and shutdown interrupts the wait.

Transient means:

  • a refused, reset or timed-out connection
  • an upstream 502/503/504 without a protocol body
  • a 503 storage_unavailable or backend_unavailable
  • any 5xx or 429 with retryable: true

Still fatal:

  • retryable: false
  • 401 and 403
  • any other 4xx
  • an incompatible Server

For these, the worker logs worker <id> stopping: registration failed with a non-retryable error: ... at error level, and run() raises.

_register itself is unchanged, so the replay-guarded symbol is untouched. Pollers, heartbeats and completions are unchanged. Docs: a new "Startup while the runtime is unavailable" section in docs/reference/worker.md.

One difference from the PHP rule: after registration, Python pollers keep retrying 401/403. See the follow-up below.

Follow-up: activity outcomes and poll auth refusals (coordinator decisions)

Activity outcomes. On main, a completion or failure report that hit a transient was retried only by the client's 3 quick attempts, then dropped with a warning (complete_error).

  • It is now resent with the shared backoff while the attempt's lease covers the next wait. That is the renewed lease when the Server supports activity lease renewal, and the claimed lease_expires_at otherwise.
  • A transient never turns a completion into a failure report.
  • When the lease runs out, the worker logs activity task <id> completed outcome not delivered after N transient failures (...): its lease ran out at error level and stops. The Server settles the attempt at lease expiry.

Duplicate completion, observed on the published durableworkflow/server:3.17.0. I put an in-process transport in front of a real worker that forwards the first /complete and then shows the client a 503 or a connection reset. The Server answers the resend with:

409 {"task_id":<same>,"activity_attempt_id":<same>,"outcome":"completed","recorded":false,"reason":"stale_attempt","next_task_id":null,"lease_owner":<this worker>,"activity_status":"completed","attempt_status":"completed","task_status":"completed","protocol_version":"1.20",...}
  • That exact shape now counts as this worker's committed completion, the same rule as PHP's isCommittedActivityCompletion. With this branch, the run completes with exactly one ActivityCompleted, the activity ran once, and the worker keeps running. That held for both the 503 and the reset cases.
  • Any other stale_attempt is a closed attempt and is never treated as success.
  • The duplicate of a committed failure report is answered 409 {"outcome":"failed","recorded":false,"reason":"stale_attempt"}, with no owner or status fields. It is reported as a closed attempt. History still has exactly one ActivityFailed.

Poll-time 401/403. Pollers still retry so rotated credentials recover without a restart; that behavior is documented. Every refusal is now logged at ERROR, not rate-limited, including refusals before a later recovery. The difference from PHP and Rust, which stop the worker, is documented in docs/reference/worker.md.

Review follow-ups: poll backoff and outage logging

  • Poll backoff: poll retry waits are now uniform(0, min(10, 0.25·2^(n−1))) seconds. That is the shared full jitter capped at 10 s, instead of a ceiling that doubled from 1 s to 30 s. A bounded Server retry delay (minimum_delay) is still honoured.
  • Long outages: a poller failing continuously for 60 s logs the outage at ERROR, then every 5 minutes while it lasts. A successful poll resets it, and the next failure starts again with a warning. Auth refusals stay ERROR on every occurrence.
  • Tests (tests/test_poll_failure_backoff.py):
    • the schedule 0.25, 0.5, 1, 2, 4, 8, 10, 10, 10, reset after a success
    • jitter bounds (0, ceiling)
    • minimum_delay is honoured
    • outage log levels: a warning at 0 s, ERROR at 61, 362 and 663 s, and a warning again after a success
  • Docs: docs/reference/worker.md is updated.

Tests (tests/test_worker_transient_runtime.py)

Added for the follow-up:

  • A completion survives 503, refused and 502 errors, with delays 0.25, 0.5 and 1 s, and no failure report.
  • A committed-duplicate resend counts as success.
  • Five other stale_attempt shapes are not success, and none of them is resent.
  • Retries stop with an error log when the lease runs out.
  • A 422 refusal is not retried.
  • A failure report is repeated through transients.
  • In tests/test_poll_failure_backoff.py: every 401/403 is logged at ERROR, including before a recovery.

From the first round:

  • Classification of 19 cases: transport errors, 502/503/504, storage_unavailable and backend_unavailable, retryable true and false, storage_pressure, 401, 403, 409, and compatibility errors.
  • The shared backoff ceilings: 0.25, 0.5, 1, 2, 4, 4 s.
  • A real Client and Worker.run() over httpx.MockTransport. Eight transient failures on discovery or registration (503 storage_unavailable, refused connection, bare 502, backend_unavailable), and two on the activity poll, are followed by recovery. The worker keeps running, registers, polls, and logs nothing at error level.
  • Fatal cases on registration or discovery: 401, 403, and 503 retryable: false. Each is served once, never retried, never polls, and logs the error-level stop line.
  • Shutdown interrupts a transient retry.

Locally (Python 3.14):

  • pytest tests/ -m "not integration": 7855 passed, 3 skipped.
  • ruff, mypy and the regression-corpus guard are clean.

🤖 Generated with Claude Code

Same issue as durable-workflow/sdk-php#448. Worker.run() and run_until()
raised once the client's three transport attempts failed on cluster
discovery or registration, so a native Server answering
503 {"reason":"storage_unavailable"} during warm-up (or a refused
connection, or an upstream 502/503/504) ended the worker.

Discovery and registration are now repeated whole through transient
failures with the shared backoff uniform(0, min(4, 0.25 * 2**(n-1))) s:
refused/reset/timed-out connections, upstream 502/503/504 without a
protocol body, 503 storage_unavailable/backend_unavailable, and 5xx/429
with retryable: true. retryable: false, 401/403, other 4xx and an
incompatible Server stay fatal and are logged at error level before
run() raises. Pollers, completions and worker heartbeats already rode
out these failures and are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel added bug Something isn't working agent:general labels Nov 20, 2026
rmcdaniel and others added 2 commits November 20, 2026 06:26
Coordinator decisions for #355:

- An activity completion or failure report that hits a transient is
  resent with the shared backoff uniform(0, min(4, 0.25 * 2**(n-1))) s
  while the attempt's lease (renewed lease, else the claimed
  lease_expires_at) covers the next wait. A transient never becomes a
  failure report. When the lease runs out the resend stops with an error
  log and the Server settles the attempt at lease expiry.
- A resend answered 409 stale_attempt, recorded false, same task/attempt
  /owner, activity/attempt/task status all completed is this worker's
  committed completion (observed on durableworkflow/server:3.17.0; same
  rule as PHP's isCommittedActivityCompletion). Any other stale_attempt
  is a closed attempt, never success.
- Poll-time 401/403 keep retrying so credentials can rotate, but every
  refusal is now logged at ERROR. The cross-SDK difference (PHP and Rust
  stop) is documented.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…RROR

Review follow-ups for #355:

- Poll retry waits are uniform(0, min(10, 0.25 * 2**(n-1))) s (the
  shared full jitter, capped at 10 s instead of growing to 30 s), still
  honouring a bounded Server retry delay.
- A poller failing continuously for 60 s logs the outage at ERROR, then
  every 5 minutes while it lasts; a successful poll resets it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
rmcdaniel added a commit that referenced this pull request Nov 20, 2026
The duplicate-completion check runs when the committed-completion rule
(#355) is present; it logs at INFO, so the test captures that level.
Verified against this branch merged with #355.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Keeps both: this branch's transient retries, committed-completion rule
and outage logs, and main's decoded response cap and continue-as-new
result. Only CHANGELOG.md conflicted; both entries are kept.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

rmcdaniel added a commit that referenced this pull request Nov 20, 2026
Brings in the committed-completion rule, so the end-to-end test's
garbled completion resend (409 stale_attempt) is accepted as success.
Only CHANGELOG.md conflicted; both entries are kept.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel
rmcdaniel merged commit 9caea2b into main Nov 20, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:general bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant